Skip to content

feat: add constructive-compute-stubs — schema-only DDL for FK resolution#51

Closed
pyramation wants to merge 100 commits into
mainfrom
feat/constructive-infra-schema
Closed

feat: add constructive-compute-stubs — schema-only DDL for FK resolution#51
pyramation wants to merge 100 commits into
mainfrom
feat/constructive-infra-schema

Conversation

@pyramation

@pyramation pyramation commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds database/constructive-compute-stubs/ — a pgpm package containing schema-only DDL (tables, columns, constraints, indexes) for 19 tables that the compute layer needs for FK resolution, with zero behavioral code.

4 schemas, 19 tables:

  • services_public (15): apps, apis, domains, sites, api_modules, api_schemas, api_settings, cors_settings, database_settings, pubkey_settings, rls_settings, site_metadata, site_modules, site_themes, webauthn_settings
  • constructive_users_public (2): users, role_types
  • constructive_infra_public (2): platform_namespaces, platform_namespace_events
  • constructive_infra_private: schema only (no tables)

What's excluded (via upstream excludeCategories in constructive-db#1590):

  • All auth tables (sessions, rate limits, identity providers) — category 'auth'
  • All membership tables (memberships, members, profiles, settings) — category 'memberships'
  • All permission tables (SPRTs, grants, permission defaults) — category 'permissions'
  • All security code (RLS policies, grants, triggers, procedures, views) — stubMode

Zero: policies, grants, triggers, trigger functions, procedures, views, or fixtures.

Link to Devin session: https://app.devin.ai/sessions/633efc205f0c445dbbb61c40679c2f78
Requested by: @pyramation

Sliced from constructive-db's monolithic constructive module using:
  pnpm run slice:constructive -- --renumber-alterations --strip-cross-package-deps

Package includes:
- constructive_infra_public schema, types, and all tables
- constructive_infra_private schema and trigger functions
- 254 changes, 764 files

Standalone: requires only plpgsql, pgpm-inflection, pgpm-stamps
No RLS policies, no grants, no job triggers
@devin-ai-integration

Copy link
Copy Markdown

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

…mignore)

The slicer now generates a complete pgpm module with:
- package.json with @pgpm/* dependencies mapped from .control requires
- Makefile with PGXS include
- .npmignore excluding test files

This fixes 'No package.json found' error when running pgpm install.
@socket-security

socket-security Bot commented Jun 8, 2026

Copy link
Copy Markdown

All alerts resolved. Learn more about Socket for GitHub.

This PR previously contained dependency changes with security issues that have been resolved, removed, or ignored.

View full report

…ema stripping

- Add missing deploy dependencies (type, constraint→column, FK→pkey)
- Plan is now topologically sorted for correct deploy order
- Remove metaschema partition registrations (INSERT INTO metaschema_public.*)
  that require the closed-source MetaSchema system
- Successfully tested: pgpm deploy --yes --createdb --database testinfra
…tion

Separate pgpm module that registers the constructive-infra function tables
as a MetaSchema function_module. This allows querying MetaSchema to
discover the infra function tables (definitions, invocations, execution_logs,
secret_definitions).

Dependencies:
  - constructive-infra (the raw DDL)
  - @pgpm/metaschema-modules (function_module table)
  - @pgpm/services (service registration layer)

The registration INSERT provides explicit schema/table names for the
platform-scoped infra tables, letting the MetaSchema trigger system
handle ID resolution and API routing.
Adds a CI workflow that runs 'pgpm test-packages --full-cycle' on every
PR that touches pgpm/ files. This validates the full deploy/verify/revert/deploy
cycle for constructive-infra against a real postgres-plus:18 instance.

constructive-infra-services is excluded since it requires the full
MetaSchema system (closed-source triggers) to deploy.

Also fixes the function_requirement type revert SQL which was missing
the TYPE keyword (DROP → DROP TYPE).
@socket-security

socket-security Bot commented Jun 8, 2026

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addednpm/​@​constructive-io/​constructive-infra@​0.0.1100100100100100

View full report

pyramation added 20 commits June 8, 2026 02:37
- job/compute-worker: platform-aware worker that discovers functions from
  constructive_infra_public.platform_function_definitions (TTL-cached),
  tracks invocations in platform_function_invocations, dispatches via HTTP
- job/compute-service: orchestrator (callback server + ComputeWorker +
  Scheduler), mirrors job/service patterns
- scripts/setup-platform-db.sh: Tier 1 setup (pgpm deploy + seed)
- scripts/dev-compute.ts: dev launcher for compute-service + functions
- scripts/seed-functions.sql: registers send-email + send-verification-link
- docker-compose.yml: adds platform-setup service (Tier 2)
- k8s/overlays/local-simple/compute-service.yaml: K8s manifest (Tier 3)
- Makefile: adds setup-platform, dev-compute targets + tier documentation
- .agents/skills/dev-tiers: skill for the 3-tier dev model
- .agents/skills/compute-worker: skill for the compute-worker system
- pgpm/constructive-infra: adds @pgpm/database-jobs dependency
…error)

pgpm deploy requires module dependencies to be pre-installed via
'pgpm install' before deployment. Without this, packages like
@pgpm/database-jobs fail with 'extension not available'.

Adds pgpm install to:
- scripts/setup-platform-db.sh (Tier 1)
- docker-compose.yml platform-setup service (Tier 2)
- .github/workflows/pgpm-test.yaml (CI)
- make status: shows Docker containers, PG connection, databases with
  infra schema, Node/pnpm/pgpm versions, and build state
- make verify-platform: checks DB exists, infra + jobs schemas deployed,
  tables present, functions seeded — exits non-zero with fix instructions
  if anything is wrong
Dan committed all extension deps (database-jobs, metaschema-modules,
services, etc.) directly into extensions/. pgpm install is no longer
needed before deploy. Also restores pgpm volume to :ro in Docker
Compose and mounts extensions/ for the platform-setup service.
Built-in function definitions (send-email, send-verification-link) are
now deployed as a pgpm fixture inside constructive-infra. The fixture
uses the standard deploy/revert/verify pattern following the inflection
module's fixtures convention.

- deploy: INSERT ON CONFLICT DO NOTHING
- revert: DELETE WHERE is_built_in AND scope='platform'
- verify: SELECT 1 from seed row

Removes manual psql seed step from setup script, docker-compose, and
verify-platform fix instructions. pgpm deploy now handles everything.
Moves the built-in function seed data (send-email, send-verification-link)
from constructive-infra into its own pgpm package: constructive-infra-seed.

This keeps constructive-infra as pure DDL (schemas, tables, triggers) and
the seed data as a separate deployable unit. The seed package depends on
constructive-infra via the .control file requires.

Both packages pass pgpm test-packages --full-cycle.
compute-service discovers functions from the database, so it doesn't
need direct workspace deps on send-email-fn or send-verification-link-fn
(which are generated packages in generated/ and not in the workspace).
…nfigs

- New fixture: seed_built_in_secrets seeds MAILGUN_* and SMTP_* into
  platform_secret_definitions (with well-known default database_id)
- Function definitions now include required_secrets and required_configs
  arrays (function_requirement[] type) linking functions to their deps
- All SQL files now use proper pgpm format: BEGIN/COMMIT wrappers,
  '-- Deploy:' header style with '-- made with <3 @ constructive.io'
- New script: load-platform-env.sh reads .env, cross-references against
  DB function requirements, reports satisfied vs missing keys
- .env.example updated with SMTP/Mailpit + dry-run toggle sections
- Makefile: added 'make check-env' target
Procedural lifecycle targets:
  make up                  # prereqs → docker → bootstrap → deploy → seed → verify
  make up DB_NAME=mydb     # same with custom DB
  make down                # stop docker compose + pgpm docker stop
  DROP=1 make down DB_NAME=mydb  # also drop the DB
  make up:email-job        # start mailpit + compute-service (SMTP mode)
  make down:email-job      # stop mailpit + compute-service

email-job-up verifies platform is up first, starts mailpit, loads
.env (with sane SMTP defaults), then launches compute-service.

status now shows mailpit container state.
…rrides

- Skip python-example and other non-node-graphql functions from startup
- Print a clear port/service summary table before launching processes
- Respect SEND_EMAIL_DRY_RUN, SEND_VERIFICATION_LINK_DRY_RUN,
  EMAIL_SEND_USE_SMTP, and SMTP_FROM from environment (was hardcoded)
- www/: Vite + React + Express app with 6 tabs (Functions, Secrets, Jobs,
  Invocations, Commands, Terminal)
- Express backend: REST API for DB queries + WebSocket terminal via child_process
- Secrets & Namespaces tab: shows seeded secret definitions and default namespace
- Seed: add default platform namespace, link functions to namespace_id
- Function API: parse composite-type arrays into proper JSON
- Makefile: add make up:www target
- scripts/www-up.sh: checks platform, installs deps, starts Vite + Express
…ster management

- Install commander and kubernetesjs packages in www/
- Add K8s proxy endpoint (/api/k8s/*) in Express server
- Add K8s tab with Pods/Deployments/Services views per namespace
- Namespace selector defaults to constructive-functions
- Graceful error when kubectl proxy not running
… tab

- ansiToHtml() converts ANSI escape sequences into colored <span> elements
- Remove max-h-60 so output expands naturally with content
- Exit code shown in a separate footer bar
- Strip remaining escape sequences that don't match known codes
Step 4b cross-references loaded env vars against platform_function_definitions
required_secrets/required_configs. Warns about missing secrets but doesn't block
(defaults still work for Mailpit/SMTP mode). Suggests 'make check-env' for details.
- Reads .env file and merges with process.env + dev defaults
- Queries platform_function_definitions for required_secrets/required_configs
- Reports per-function coverage at startup (✓ all set / ● N missing)
- Priority: .env > process.env > hardcoded defaults
- Graceful fallback if DB query fails (schema not deployed)
Both tables are PARTITION BY RANGE(created_at) but had no partitions,
causing 'no partition of relation found for row' on INSERT.
Adds DEFAULT partition for each so rows land somewhere until
time-based partitions are created.
Secrets tab now shows:
- Function coverage badges (e.g. send-email 5/8)
- All required secrets/configs from DB + .env merged
- Editable input fields with eye/reveal toggle for sensitive keys
- 'Save to .env' button writes grouped .env file to project root
- Shows which functions require each secret
- File status indicator (.env exists / will be created)

Backend adds:
- GET /api/env — reads .env and returns parsed vars
- POST /api/env — merges values + writes grouped .env file
Each function card now has a 'Trigger' button that opens an inline
payload editor with sensible defaults (send-email gets to/subject/html,
send-verification-link gets to/type/link). Submitting creates a job in
app_jobs.jobs and shows success/error inline — no need to switch tabs.
fix: auto-generate function packages when missing
…efinitions

- scripts/dev-compute.ts: fix DEFAULT_DATABASE_ID from 'dbe' to proper nil UUID,
  update function_definitions query to use constructive_compute_public
- scripts/email-job-up.sh: check for constructive_compute_public schema,
  query function_definitions from constructive_compute_public
- scripts/register-functions.ts: INSERT into constructive_compute_public
- scripts/seed-functions.sql: INSERT into constructive_compute_public
- scripts/status.sh: check constructive_compute_public for function counts
- scripts/setup-platform-db.sh: verify function count from correct schema
- scripts/secrets-sync.sh: query function requirements from correct schema
- scripts/load-platform-env.sh: query function definitions from correct schema

platform_namespaces remains in constructive_infra_public (not moved).
fix: update scripts to use constructive_compute_public for function definitions
- module-loader: catch missing function_invocation_module table gracefully
- discovery: fall back to constructive_compute_public when function_module
  config not populated in metaschema
- compute-service: fall back to direct schema check in waitForComputePrereqs
  when metaschema database row doesn't exist
Updates all @pgpm dependency packages including metaschema-modules
which now includes the function_prefix column on merkle_store_module.
Replaces constructive-compute-fbp with constructive-platform-function-graph
(unprefixed merkle store functions: get_all, get_node_at_path, etc.)

Updates constructive-compute with FBP operations now consolidated into
constructive_compute_public schema.
…platform-function-graph

- up.sh: deploy constructive-platform-function-graph instead of constructive-compute-fbp
- verify-platform.sh: check new schema names
- infra-seed: update partition tables to constructive_compute_private
- copy-slicer-packages.py: update package mapping
Replaced by constructive-platform-function-graph (merkle store functions)
and constructive-compute (FBP operations) after schema consolidation.
…-update

feat: unprefixed functions + schema consolidation + graceful fallback
The platform_files_file_path and platform_files_rename functions reference
columns (filename, id) via composite type parameters. PostgreSQL validates
SQL function bodies at creation time, so the columns must exist before the
functions are created.

Moved procedure entries in pgpm.plan to after all column additions and added
explicit column dependencies in the deploy SQL requires headers.
fix: reorder pgpm.plan so storage procedures deploy after columns
… RLS/partman

Fixes all pgpm deploy errors for standalone (non-monolith) mode:

1. pgpm.plan ordering: Move procedures/trigger_fns that reference table
   composite types to AFTER the table definitions they depend on.
   Affects: constructive-objects, constructive-platform-function-graph,
   constructive-compute.

2. Remove constructive-compute-stubs: Replace all stubs references with
   constructive-infra (the actual module). Update .control files and
   pgpm.plan cross-package deps. Delete the stubs directory entirely.

3. Strip RLS policies: Remove enable_row_level_security and auth_*
   policy entries from objects/storage/store pgpm.plan files — these
   reference jwt_public and constructive_memberships schemas not
   available in standalone mode.

4. Strip jwt_public triggers: Remove force_current_user_actor_id
   trigger_fns and their triggers from storage (depend on jwt_public).

5. Strip partman entries: Remove metaschema_public.partition INSERTs
   from compute pgpm.plan (metaschema not available standalone).

6. Strip monolith trigger: Remove platform_function_definitions job
   trigger that references the constructive monolith package.

7. Fix seed data: Update partition seed to use platform_function_*
   table names (was app_function_*). Remove stale invocations_table_name
   column from function_module registration. Fix verify script table name.
fix: pgpm deploy standalone — reorder procedures, remove stubs, strip RLS/partman
…e_compute_public

After schema consolidation, platform_function_definitions,
platform_function_invocations, and function_requirement type moved
from constructive_infra_public to constructive_compute_public.

Updates: www/server/index.ts, scripts/register-functions.ts,
docs/spec/fbp-integration.md, compute-worker + fbp skill files.
…-schema-refs

fix: update schema refs from constructive_infra_public to constructive_compute_public
New pgpm package that seeds the full API routing chain for standalone mode:
- database row (well-known 00000000-...0000 sentinel)
- 12 schema registrations (7 public + 5 private)
- 2 table registrations (definitions + secrets) for module FK refs
- compute API linked to all 7 public schemas
- compute.localhost domain
- app.localhost site + site domain

Also updates constructive-infra-services to properly resolve schema_id,
private_schema_id, definitions_table_id, and secret_definitions_table_id
via JOINs against the seeded metaschema rows.

up.sh gains step 5 (platform-seed) before step 6 (infra-seed + services).
feat: add constructive-platform-seed (metaschema routing chain)
Adds step 8 to up.sh: starts @constructive-io/graphql-server as a
background process using the sibling constructive repo, with PID file
for clean shutdown.

- start-graphql-server.sh: resolves ../constructive, starts server with
  enableServicesApi=true + isPublic=true, writes PID to .graphql-server.pid
- up.sh: step 8 starts the server, prints compute.localhost URLs
- down.sh: kills the server via PID file before stopping MinIO/PG

The server uses the services_public routing chain seeded by
constructive-platform-seed to resolve compute.localhost -> 7 schemas.

Total steps: 9 -> 10.
feat: integrate GraphQL server into make up/down (port 6464)
- Compute API (compute.localhost): users, infra, objects, storage, store, compute
- Graph API (graph.localhost): platform_function_graph, compute

Prevents PostGraphile function name collisions between
constructive_objects_public and constructive_platform_function_graph_public
(both export unprefixed merkle store functions: init_empty_repo, get_all, etc.)
feat: split compute and graph into separate APIs
Restructure seed to match the canonical API-to-schema mapping from
constructive-db's introspection.provision_base_modules:

- api: users, infra, storage, store (general platform schemas)
- compute: compute_public + platform_function_graph_public
  (matches COMPUTE_SCHEMAS = [compute_public, compute_fbp_public])
- objects: objects_public (content-addressed merkle store)
- flow: empty placeholder (populated at runtime in constructive-db)

Previously objects was lumped into compute, causing PostGraphile
collisions between the objects and graph merkle store functions.
Now each API has its own domain: api/compute/objects/flow.localhost.
feat: align API routing with constructive-db provision_base_modules
feat: add make kill (down + pgpm kill)
Adds 4 SDK packages matching constructive-db's structure:

- constructive-functions-schema: exports .graphql from live endpoints
  (api, compute, objects) via introspection
- constructive-functions-sdk: generates typed ORM client from schema files
- constructive-functions-cli: generates unified CLI from schema files
- constructive-functions-hooks: generates React Query hooks for www/

Includes:
- Root-level generate scripts (generate:schemas, generate:sdk, etc.)
- Makefile targets (make generate:schemas, make generate:sdk-all)
- CI workflow (.github/workflows/generate-sdk.yaml) for automated regen
- Auto-generated .agents/skills/ for ORM, CLI, and hooks reference
- All generated output from current schema (3 APIs, 25 tables total)
feat: SDK generation pipeline (schemas, ORM, CLI, hooks)
@devin-ai-integration

Copy link
Copy Markdown

Superseded by #95 — includes constructive-compute-stubs, then replaced with per-module packages, plus infra-seed, platform-seed, and infra-services.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant